import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
import torch
import torchvision.datasets
import torchvision.models
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
train_data = torchvision.datasets.FashionMNIST(
root="./data",
train=True,
download=True,
transform=transforms.ToTensor() # Converts to tensor but does NOT normalize
)
test_data = torchvision.datasets.FashionMNIST(
root="./data",
train=False,
download=True,
transform=transforms.ToTensor()
)Restricted Boltzmann Machines
Theory of RBMs and Applications
April 10, 2025
Restricted Boltzmann Machines
- Boltzmann machines can have hidden units, which are not directly observed or specified and allow the model to learn latent features of the data
- The Restricted Boltzmann Machine (RBM) is a special case where there are no connections between visible units or between hidden units, which allows for efficient training and sampling
(Figure 1)
Background on Models for Classification Task
We train Logistic Regression (with and without RBM features as input), Feed Forward Network (with and without RBM features as input), and Convolutional Neural Network.
For the models incoroporating the RBM, we take the Fashion MNIST features/pixels and train the RBM (unsupervised learning) to extract hidden features from the visible layer and then feed these features into either logistic regression or feed forward network. We then use the trained model to predict labels for the test data, evaluating how well the RBM-derived features perform in a supervised classification task.
1. Logistic Regression
Mathematically, the concept behind binary logistic regression is the logit (the natural logarithm of an odds ratio)(Peng, Lee, and Ingersoll 2002). However, since we have 10 labels, our classification task falls into “Multinomial Logistic Regression.”
\[ P(Y = k | X) = \frac{e^{\beta_{0k} + \beta_k^T X}}{\sum_{l=1}^{K} e^{\beta_{0l} + \beta_l^T X}} \qquad(2)\]
2. Simple Feed Forward Neural Network
The feed forward network (FNN) is one where information flows in one direction from input to output with no loops or feedback. There can be zero hidden layers in between (called single FNN) or one or more hidden layers (multilayer FNN).(Sazlı 2006)
(Figure 2)
3. Convolutional Neural Network
The convolutional neural network (CNN) is a type of feed forward network except that unlike the traditional ANN, CNNs are primarily used for pattern recognition with images (O’Shea and Nash 2015). The CNN has 3 layers which are stacked to form the full CNN: convolutional, pooling, and fully-connected layers.
(Figure 3)
Analysis and Results
Data Exploration and Visualization
We use the Fashion MNIST dataset from Zalando Research (Xiao, Rasul, and Vollgraf 2017). The set includes 70,000 grayscale images of clothing items, 60,000 for training and 10,000 for testing. Each image is 28x28 pixels (784 pixels total). Each pixel has a value associated with it ranging from 0 (white) to 255 (very dark) – whole numbers only. There are 785 columns in total as one column is dedicated to the label.
(Figure 4)
There are 10 labels in total:
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot
Click to Show Code and Output
Click to Show Code and Output
# Extract the first image (or choose any index)
image_tensor, label = train_data[6] # shape: [1, 28, 28]
# Convert tensor to NumPy array
image_array = image_tensor.numpy().squeeze()
# Plot the image
plt.figure(figsize=(5,5))
plt.imshow(image_array, cmap="gray")
plt.title(f"FashionMNIST Image (Label: {label})")
plt.axis("off") # Hide axes
plt.show()
(Figure 5)
Click to Show Code and Output
Click to Show Code and Output
#print(train_images[:5])
flattened = train_images[:5].reshape(5, -1)
# Create a DataFrame
df_flat = pd.DataFrame(flattened)
print(df_flat.head())
#train_df.info() #datatypes are integers 0 1 2 3 4 5 6 7 8 9 ... 774 775 776 777 \
0 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0
1 0 0 0 0 0 1 0 0 0 0 ... 119 114 130 76
2 0 0 0 0 0 0 0 0 0 22 ... 0 0 1 0
3 0 0 0 0 0 0 0 0 33 96 ... 0 0 0 0
4 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0
778 779 780 781 782 783
0 0 0 0 0 0 0
1 0 0 0 0 0 0
2 0 0 0 0 0 0
3 0 0 0 0 0 0
4 0 0 0 0 0 0
[5 rows x 784 columns]Click to Show Code and Output
unique_labels, counts = np.unique(train_labels, return_counts=True)
# Print the counts sorted by label
for label, count in zip(unique_labels, counts):
print(f"Label {label}: {count}")Label 0: 6000
Label 1: 6000
Label 2: 6000
Label 3: 6000
Label 4: 6000
Label 5: 6000
Label 6: 6000
Label 7: 6000
Label 8: 6000
Label 9: 6000t-SNE Visualization of Classes
t-SNE Visualizationt-distributed Stochastic Neighbor Embedding (t-SNE) is used here to visualize the separation between classes in a high-dimensional dataset.
Each point represents a single fashion item (e.g., T-shirt, Trouser, etc.), and the color corresponds to its true label across the 10 categories listed above.
Click to Show Code and Output
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
# Run t-SNE to reduce dimensionality
#embeddings = TSNE(n_jobs=2).fit_transform(X)
tsne = TSNE(n_jobs=-1, random_state=42) # Use -1 to use all available cores
embeddings = tsne.fit_transform(X) #use scikitlearn instead
# Create scatter plot
figure = plt.figure(figsize=(15,7))
plt.scatter(embeddings[:, 0], embeddings[:, 1], c=train_labels,
cmap=plt.cm.get_cmap("jet", 10), marker='.')
plt.colorbar(ticks=range(10))
plt.clim(-0.5, 9.5)
plt.title("t-SNE Visualization of Fashion MNIST")
plt.show()
(Figure 6)
What the visualization shows:
Class 1 (blue / Trousers) forms a clearly distinct and tightly packed cluster, indicating that the pixel patterns for trousers are less similar to those of other classes. In contrast, Classes 4 (Coat), 6 (Shirt), and 2 (Pullover) show significant overlap, suggesting that these clothing items are harder to distinguish visually and may lead to more confusion during classification.
Model 1
Model 1: Logistic Regression on Fashion MNIST Data
Click to Show Code and Output
from sklearn.metrics import f1_score
CLASSIFIER = "LogisticRegression" # Change for FNN, LogisticRegression, or CNN
# Define CNN model
class FashionCNN(nn.Module):
def __init__(self, filters1, filters2, kernel1, kernel2):
super(FashionCNN, self).__init__()
self.layer1 = nn.Sequential(
nn.Conv2d(in_channels=1, out_channels=filters1, kernel_size=kernel1, padding=1),
nn.BatchNorm2d(filters1),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.layer2 = nn.Sequential(
nn.Conv2d(in_channels=filters1, out_channels=filters2, kernel_size=kernel2),
nn.BatchNorm2d(filters2),
nn.ReLU(),
nn.MaxPool2d(2)
)
self.fc1 = None #initialize first fully connected layer as none, defined later in fwd
self.drop = nn.Dropout2d(0.25)
self.fc2 = nn.Linear(in_features=600, out_features=120)
self.fc3 = nn.Linear(in_features=120, out_features=10)
def forward(self, x):
out = self.layer1(x)
out = self.layer2(out)
#Flatten tensor dynamically, preserve batch size
out = out.view(out.size(0), -1)
if self.fc1 is None:
self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)
out = self.fc1(out)
out = self.drop(out)
out = self.fc2(out)
out = self.fc3(out)
return out
# Define Optuna objective function
def objective(trial):
# Set MLflow experiment name
if CLASSIFIER == "LogisticRegression":
experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-noRBM")
elif CLASSIFIER == "FNN":
experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-noRBM")
elif CLASSIFIER == "CNN":
experiment = mlflow.set_experiment("new-pytorch-fmnist-cnn-noRBM")
batch_size = trial.suggest_int("batch_size", 64, 256, step=32)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
mlflow.start_run(experiment_id=experiment.experiment_id)
num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)
mlflow.log_param("num_classifier_epochs", num_classifier_epochs)
if CLASSIFIER == "FNN":
hidden_size = trial.suggest_int("fnn_hidden", 192, 384)
learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)
mlflow.log_param("classifier", "FNN")
mlflow.log_param("fnn_hidden", hidden_size)
mlflow.log_param("learning_rate", learning_rate)
model = nn.Sequential(
nn.Linear(784, hidden_size),
nn.ReLU(),
nn.Linear(hidden_size, 10)
).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
elif CLASSIFIER == "CNN":
filters1 = trial.suggest_int("filters1", 16, 64, step=16)
filters2 = trial.suggest_int("filters2", 32, 128, step=32)
kernel1 = trial.suggest_int("kernel1", 3, 5)
kernel2 = trial.suggest_int("kernel2", 3, 5)
learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)
mlflow.log_param("classifier", "CNN")
mlflow.log_param("filters1", filters1)
mlflow.log_param("filters2", filters2)
mlflow.log_param("kernel1", kernel1)
mlflow.log_param("kernel2", kernel2)
mlflow.log_param("learning_rate", learning_rate)
model = FashionCNN(filters1, filters2, kernel1, kernel2).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
elif CLASSIFIER == "LogisticRegression":
mlflow.log_param("classifier", "LogisticRegression")
# Prepare data for Logistic Regression (Flatten 28x28 images to 784 features)
train_features = train_dataset.data.view(-1, 784).numpy()
train_labels = train_dataset.targets.numpy()
test_features = test_dataset.data.view(-1, 784).numpy()
test_labels = test_dataset.targets.numpy()
# Normalize the pixel values to [0,1] for better convergence
train_features = train_features / 255.0
test_features = test_features / 255.0
C = trial.suggest_float("C", 0.01, 10.0, log=True)
solver = "saga"
model = LogisticRegression(C=C, max_iter=num_classifier_epochs, solver=solver)
model.fit(train_features, train_labels)
predictions = model.predict(test_features)
accuracy = accuracy_score(test_labels, predictions) * 100
macro_f1 = f1_score(test_labels, predictions, average="macro") #for f1
print(f"Logistic Regression Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
mlflow.log_param("C", C)
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1) #for f1
mlflow.end_run()
return accuracy
# Training Loop for FNN and CNN
criterion = nn.CrossEntropyLoss()
model.train()
for epoch in range(num_classifier_epochs):
running_loss = 0.0
for images, labels in train_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
optimizer.zero_grad()
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
print(f"{CLASSIFIER} Epoch {epoch+1}: loss = {running_loss / len(train_loader):.4f}")
# Model Evaluation
model.eval()
correct, total = 0, 0
all_preds = [] # for f1
all_labels = []
with torch.no_grad():
for images, labels in test_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
_, predicted = torch.max(outputs, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
all_preds.extend(predicted.cpu().numpy()) #for f1
all_labels.extend(labels.cpu().numpy()) #for f1
accuracy = 100 * correct / total
macro_f1 = f1_score(all_labels, all_preds, average="macro") #for f1
print(f"Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1) #for f1
mlflow.end_run()
return accuracy
if __name__ == "__main__":
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=1) # n_trials set to 1 for quick rendering
print(f"Best Parameters for {CLASSIFIER}:", study.best_params)
print("Best Accuracy:", study.best_value)
(Figure 7)
\[ C = \frac{1}{\lambda} \quad \text{(inverse regularization strength)} \]
Lower values of C mean more regularization (higher penalties for larger weight coefficients)
What the plot shows:
Most optuna trials were lower values of C, so optimization favors stronger regularization. This is further evidenced by the clustering of higher accuracies for lower values of C. A possibly anomaly is seen at C=10 with fairly high accuracy; however, it’s still not higher than lower values of C.
Model 2
Model 2: Feed Forward Network on Fashion MNIST Data
Click to Show Code and Output
from sklearn.metrics import f1_score
CLASSIFIER = "FNN" # Change for FNN, LogisticRegression, or CNN
# Define CNN model
class FashionCNN(nn.Module):
def __init__(self, filters1, filters2, kernel1, kernel2):
super(FashionCNN, self).__init__()
self.layer1 = nn.Sequential(
nn.Conv2d(in_channels=1, out_channels=filters1, kernel_size=kernel1, padding=1),
nn.BatchNorm2d(filters1),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.layer2 = nn.Sequential(
nn.Conv2d(in_channels=filters1, out_channels=filters2, kernel_size=kernel2),
nn.BatchNorm2d(filters2),
nn.ReLU(),
nn.MaxPool2d(2)
)
self.fc1 = None #initialize first fully connected layer as none, defined later in fwd
self.drop = nn.Dropout2d(0.25)
self.fc2 = nn.Linear(in_features=600, out_features=120)
self.fc3 = nn.Linear(in_features=120, out_features=10)
def forward(self, x):
out = self.layer1(x)
out = self.layer2(out)
#Flatten tensor dynamically
out = out.view(out.size(0), -1)
if self.fc1 is None:
self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)
out = self.fc1(out)
out = self.drop(out)
out = self.fc2(out)
out = self.fc3(out)
return out
# Define Optuna objective function
def objective(trial):
# Set MLflow experiment name
if CLASSIFIER == "LogisticRegression":
experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-noRBM")
elif CLASSIFIER == "FNN":
experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-noRBM")
elif CLASSIFIER == "CNN":
experiment = mlflow.set_experiment("new-pytorch-fmnist-cnn-noRBM")
batch_size = trial.suggest_int("batch_size", 64, 256, step=32)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
mlflow.start_run(experiment_id=experiment.experiment_id)
num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)
mlflow.log_param("num_classifier_epochs", num_classifier_epochs)
if CLASSIFIER == "FNN":
hidden_size = trial.suggest_int("fnn_hidden", 192, 384)
learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)
mlflow.log_param("classifier", "FNN")
mlflow.log_param("fnn_hidden", hidden_size)
mlflow.log_param("learning_rate", learning_rate)
model = nn.Sequential(
nn.Linear(784, hidden_size),
nn.ReLU(),
nn.Linear(hidden_size, 10)
).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
elif CLASSIFIER == "CNN":
filters1 = trial.suggest_int("filters1", 16, 64, step=16)
filters2 = trial.suggest_int("filters2", 32, 128, step=32)
kernel1 = trial.suggest_int("kernel1", 3, 5)
kernel2 = trial.suggest_int("kernel2", 3, 5)
learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)
mlflow.log_param("classifier", "CNN")
mlflow.log_param("filters1", filters1)
mlflow.log_param("filters2", filters2)
mlflow.log_param("kernel1", kernel1)
mlflow.log_param("kernel2", kernel2)
mlflow.log_param("learning_rate", learning_rate)
model = FashionCNN(filters1, filters2, kernel1, kernel2).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
elif CLASSIFIER == "LogisticRegression":
mlflow.log_param("classifier", "LogisticRegression")
# Prepare data for Logistic Regression (Flatten 28x28 images to 784 features)
train_features = train_dataset.data.view(-1, 784).numpy()
train_labels = train_dataset.targets.numpy()
test_features = test_dataset.data.view(-1, 784).numpy()
test_labels = test_dataset.targets.numpy()
# Normalize the pixel values to [0,1] for better convergence
train_features = train_features / 255.0
test_features = test_features / 255.0
C = trial.suggest_float("C", 0.01, 10.0, log=True)
solver = "saga"
model = LogisticRegression(C=C, max_iter=num_classifier_epochs, solver=solver)
model.fit(train_features, train_labels)
predictions = model.predict(test_features)
accuracy = accuracy_score(test_labels, predictions) * 100
macro_f1 = f1_score(test_labels, predictions, average="macro") #for f1
print(f"Logistic Regression Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
mlflow.log_param("C", C)
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1) #for f1
mlflow.end_run()
return accuracy
# Training Loop for FNN and CNN
criterion = nn.CrossEntropyLoss()
model.train()
for epoch in range(num_classifier_epochs):
running_loss = 0.0
for images, labels in train_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
optimizer.zero_grad()
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
print(f"{CLASSIFIER} Epoch {epoch+1}: loss = {running_loss / len(train_loader):.4f}")
# Model Evaluation
model.eval()
correct, total = 0, 0
all_preds = [] # for f1
all_labels = []
with torch.no_grad():
for images, labels in test_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
_, predicted = torch.max(outputs, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
all_preds.extend(predicted.cpu().numpy()) #for f1
all_labels.extend(labels.cpu().numpy()) #for f1
accuracy = 100 * correct / total
macro_f1 = f1_score(all_labels, all_preds, average="macro") #for f1
print(f"Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1) #for f1
mlflow.end_run()
return accuracy
if __name__ == "__main__":
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=1) # n_trials set to 1 for quick rendering
print(f"Best Parameters for {CLASSIFIER}:", study.best_params)
print("Best Accuracy:", study.best_value)
(Figure 8)
What the plot shows:
Higher values of hidden units in the feedforward network were sampled more frequently by Optuna, suggesting a preference for more complex models. However, test accuracy appears to level off between 300 and 375 hidden units, suggesting complexity reached its optimal range. Further increases in hidden units would likely not yield higher accuracy.
Model 3
Model 3: Convolutional Neural Network on Fashion MNIST Data
Base code for CNN structure borrowed from Kaggle
Click to Show Code and Output
from sklearn.metrics import f1_score
CLASSIFIER = "CNN" # Change for FNN, LogisticRegression, or CNN
# Define CNN model
class FashionCNN(nn.Module):
def __init__(self, filters1, filters2, kernel1, kernel2):
super(FashionCNN, self).__init__()
self.layer1 = nn.Sequential(
nn.Conv2d(in_channels=1, out_channels=filters1, kernel_size=kernel1, padding=1),
nn.BatchNorm2d(filters1),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.layer2 = nn.Sequential(
nn.Conv2d(in_channels=filters1, out_channels=filters2, kernel_size=kernel2),
nn.BatchNorm2d(filters2),
nn.ReLU(),
nn.MaxPool2d(2)
)
self.fc1 = None #initialize first fully connected layer as none, defined later in fwd
self.drop = nn.Dropout2d(0.25)
self.fc2 = nn.Linear(in_features=600, out_features=120)
self.fc3 = nn.Linear(in_features=120, out_features=10)
def forward(self, x):
out = self.layer1(x)
out = self.layer2(out)
#Flatten tensor dynamically
out = out.view(out.size(0), -1)
if self.fc1 is None:
self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)
out = self.fc1(out)
out = self.drop(out)
out = self.fc2(out)
out = self.fc3(out)
return out
# Define Optuna objective function
def objective(trial):
# Set MLflow experiment name
if CLASSIFIER == "LogisticRegression":
experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-noRBM")
elif CLASSIFIER == "FNN":
experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-noRBM")
elif CLASSIFIER == "CNN":
experiment = mlflow.set_experiment("new-pytorch-fmnist-cnn-noRBM")
batch_size = trial.suggest_int("batch_size", 64, 256, step=32)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
mlflow.start_run(experiment_id=experiment.experiment_id)
num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)
mlflow.log_param("num_classifier_epochs", num_classifier_epochs)
if CLASSIFIER == "FNN":
hidden_size = trial.suggest_int("fnn_hidden", 192, 384)
learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)
mlflow.log_param("classifier", "FNN")
mlflow.log_param("fnn_hidden", hidden_size)
mlflow.log_param("learning_rate", learning_rate)
model = nn.Sequential(
nn.Linear(784, hidden_size),
nn.ReLU(),
nn.Linear(hidden_size, 10)
).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
elif CLASSIFIER == "CNN":
filters1 = trial.suggest_int("filters1", 16, 64, step=16)
filters2 = trial.suggest_int("filters2", 32, 128, step=32)
kernel1 = trial.suggest_int("kernel1", 3, 5)
kernel2 = trial.suggest_int("kernel2", 3, 5)
learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)
mlflow.log_param("classifier", "CNN")
mlflow.log_param("filters1", filters1)
mlflow.log_param("filters2", filters2)
mlflow.log_param("kernel1", kernel1)
mlflow.log_param("kernel2", kernel2)
mlflow.log_param("learning_rate", learning_rate)
model = FashionCNN(filters1, filters2, kernel1, kernel2).to(device)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
elif CLASSIFIER == "LogisticRegression":
mlflow.log_param("classifier", "LogisticRegression")
# Prepare data for Logistic Regression (Flatten 28x28 images to 784 features)
train_features = train_dataset.data.view(-1, 784).numpy()
train_labels = train_dataset.targets.numpy()
test_features = test_dataset.data.view(-1, 784).numpy()
test_labels = test_dataset.targets.numpy()
# Normalize the pixel values to [0,1] for better convergence
train_features = train_features / 255.0
test_features = test_features / 255.0
C = trial.suggest_float("C", 0.01, 10.0, log=True)
solver = "saga"
model = LogisticRegression(C=C, max_iter=num_classifier_epochs, solver=solver)
model.fit(train_features, train_labels)
predictions = model.predict(test_features)
accuracy = accuracy_score(test_labels, predictions) * 100
macro_f1 = f1_score(test_labels, predictions, average="macro") #for f1
print(f"Logistic Regression Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
mlflow.log_param("C", C)
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1) #for f1
mlflow.end_run()
return accuracy
# Training Loop for FNN and CNN
criterion = nn.CrossEntropyLoss()
model.train()
for epoch in range(num_classifier_epochs):
running_loss = 0.0
for images, labels in train_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
optimizer.zero_grad()
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
print(f"{CLASSIFIER} Epoch {epoch+1}: loss = {running_loss / len(train_loader):.4f}")
# Model Evaluation
model.eval()
correct, total = 0, 0
all_preds = [] # for f1
all_labels = []
with torch.no_grad():
for images, labels in test_loader:
images, labels = images.to(device), labels.to(device)
outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
_, predicted = torch.max(outputs, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
all_preds.extend(predicted.cpu().numpy()) #for f1
all_labels.extend(labels.cpu().numpy()) #for f1
accuracy = 100 * correct / total
macro_f1 = f1_score(all_labels, all_preds, average="macro") #for f1
print(f"Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1) #for f1
mlflow.end_run()
return accuracy
if __name__ == "__main__":
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=1) # n_trials set to 1 for quick rendering
print(f"Best Parameters for {CLASSIFIER}:", study.best_params)
print("Best Accuracy:", study.best_value)
(Figure 9)
What the plot shows:
Although the highest test accuracy was achieved with 64 filters in the first convolutional 2D layer, the number of filters alone isn’t a strong predictor of model performance. Each filter size shows high variance (accuracies are spread out for each value vertically). This, combined with the fact that accuracies are well distributed across the different filter counts, suggests other factors or hyperparameters may play a bigger role in predicting accuracy.
(Figure 10)
What the plot shows:
Like the first Conv2D layer, the number of filters doesn’t seem to be a extremely strong predictor in accuracy. However, Optuna has sampled more frequently from higher number of filters, even 128 for this second layer, suggesting higher filters performed better. However, like before, there is still high variance in accuracy for each number of filters.
(Figure 11)
What the plot shows:
Kernel size of 3 was sampled more frequently by Optuna and yielded higher accuracies than kernel sizes of 4 or 5.
(Figure 12)
What the plot shows:
Like with the first convolutional 2D layer, kernel size of 3 is highly favored by Optuna and consistently led to higher test accuracies.
Model 4
Model 4: Logistic Regression on RBM Hidden Features (of Fashion MNIST Data)
Click to Show Code and Output
from sklearn.metrics import accuracy_score, f1_score
CLASSIFIER = 'LogisticRegression'
if CLASSIFIER == 'LogisticRegression':
experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-withrbm")
else:
experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-withrbm")
class RBM(nn.Module):
def __init__(self, n_visible=784, n_hidden=256, k=1):
super(RBM, self).__init__()
self.n_visible = n_visible
self.n_hidden = n_hidden
# Initialize weights and biases
self.W = nn.Parameter(torch.randn(n_hidden, n_visible) * 0.1)
self.v_bias = nn.Parameter(torch.zeros(n_visible))
self.h_bias = nn.Parameter(torch.zeros(n_hidden))
self.k = k # CD-k steps
def sample_h(self, v):
# Given visible v, sample hidden h
p_h = torch.sigmoid(F.linear(v, self.W, self.h_bias)) # p(h=1|v)
h_sample = torch.bernoulli(p_h) # sample Bernoulli
return p_h, h_sample
def sample_v(self, h):
# Given hidden h, sample visible v
p_v = torch.sigmoid(F.linear(h, self.W.t(), self.v_bias)) # p(v=1|h)
v_sample = torch.bernoulli(p_v)
return p_v, v_sample
def forward(self, v):
# Perform k steps of contrastive divergence starting from v
v_k = v.clone()
for _ in range(self.k):
_, h_k = self.sample_h(v_k) # sample hidden from current visible
_, v_k = self.sample_v(h_k) # sample visible from hidden
return v_k # k-step reconstructed visible
def free_energy(self, v):
# Compute the visible bias term for each sample in the batch
vbias_term = (v * self.v_bias).sum(dim=1) # shape: [batch_size]
# Compute the activation of the hidden units
wx_b = F.linear(v, self.W, self.h_bias) # shape: [batch_size, n_hidden]
# Compute the hidden term
hidden_term = torch.sum(torch.log1p(torch.exp(wx_b)), dim=1) # shape: [batch_size]
# Return the mean free energy over the batch
return - (vbias_term + hidden_term).mean()
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.FashionMNIST(root='./data', train=True, transform=transform, download=True)
test_dataset = datasets.FashionMNIST(root='./data', train=False, transform=transform, download=True)
def objective(trial):
num_rbm_epochs = trial.suggest_int("num_rbm_epochs", 5, 5)# 24, 33)
batch_size = trial.suggest_int("batch_size", 192, 1024)
rbm_lr = trial.suggest_float("rbm_lr", 0.05, 0.1)
rbm_hidden = trial.suggest_int("rbm_hidden", 384, 8192)
mlflow.start_run(experiment_id=experiment.experiment_id)
if CLASSIFIER != 'LogisticRegression':
fnn_hidden = trial.suggest_int("fnn_hidden", 192, 384)
fnn_lr = trial.suggest_float("fnn_lr", 0.0001, 0.0025)
mlflow.log_param("fnn_hidden", fnn_hidden)
mlflow.log_param("fnn_lr", fnn_lr)
num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)# 40, 60)
mlflow.log_param("num_rbm_epochs", num_rbm_epochs)
mlflow.log_param("batch_size", batch_size)
mlflow.log_param("rbm_lr", rbm_lr)
mlflow.log_param("rbm_hidden", rbm_hidden)
mlflow.log_param("num_classifier_epochs", num_classifier_epochs)
# Instantiate RBM and optimizer
device = torch.device("mps")
rbm = RBM(n_visible=784, n_hidden=rbm_hidden, k=1).to(device)
optimizer = torch.optim.SGD(rbm.parameters(), lr=rbm_lr)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
rbm_training_failed = False
# Training loop (assuming train_loader yields batches of images and labels)
for epoch in range(num_rbm_epochs):
total_loss = 0.0
for images, _ in train_loader:
# Flatten images and binarize
v0 = images.view(-1, 784).to(rbm.W.device) # shape [batch_size, 784]
v0 = torch.bernoulli(v0) # sample binary input
vk = rbm(v0) # k-step CD reconstruction
# Compute contrastive divergence loss (free energy difference)
loss = rbm.free_energy(v0) - rbm.free_energy(vk)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch+1}: avg free-energy loss = {total_loss/len(train_loader):.4f}")
if np.isnan(total_loss):
rbm_training_failed = True
break
if rbm_training_failed:
accuracy = 0.0
macro_f1 = 0.0
print("RBM training failed — returning 0.0 for accuracy and macro F1")
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1)
mlflow.set_tag("status", "rbm_failed") # Optional tag
mlflow.end_run()
return float(accuracy)
else:
rbm.eval() # set in evaluation mode if using any layers that behave differently in training
features_list = []
labels_list = []
for images, labels in train_loader:
v = images.view(-1, 784).to(rbm.W.device)
v = v # (optionally binarize or use raw normalized pixels)
h_prob, h_sample = rbm.sample_h(v) # get hidden activations
features_list.append(h_prob.cpu().detach().numpy())
labels_list.append(labels.numpy())
train_features = np.concatenate(features_list) # shape: [N_train, n_hidden]
train_labels = np.concatenate(labels_list)
# Convert pre-extracted training features and labels to tensors and create a DataLoader
train_features_tensor = torch.tensor(train_features, dtype=torch.float32)
train_labels_tensor = torch.tensor(train_labels, dtype=torch.long)
train_feature_dataset = torch.utils.data.TensorDataset(train_features_tensor, train_labels_tensor)
train_feature_loader = torch.utils.data.DataLoader(train_feature_dataset, batch_size=batch_size, shuffle=True)
if CLASSIFIER == 'LogisticRegression':
# add optuna tuning same as log reg without RBM features...
lr_C = trial.suggest_float("lr_C", 0.01, 10.0, log=True)
mlflow.log_param("lr_C", lr_C) # Log the chosen C value
classifier = LogisticRegression(max_iter=num_classifier_epochs, C=lr_C, solver="saga")
classifier.fit(train_features, train_labels)
else:
classifier = nn.Sequential(
nn.Linear(rbm.n_hidden, fnn_hidden),
nn.ReLU(),
nn.Linear(fnn_hidden, 10)
)
# Move classifier to the same device as the RBM
classifier = classifier.to(device)
criterion = nn.CrossEntropyLoss()
classifier_optimizer = torch.optim.Adam(classifier.parameters(), lr=fnn_lr)
classifier.train()
for epoch in range(num_classifier_epochs):
running_loss = 0.0
for features, labels in train_feature_loader:
features = features.to(device)
labels = labels.to(device)
# Forward pass through classifier
outputs = classifier(features)
loss = criterion(outputs, labels)
# Backpropagation and optimization
classifier_optimizer.zero_grad()
loss.backward()
classifier_optimizer.step()
running_loss += loss.item()
avg_loss = running_loss / len(train_feature_loader)
print(f"Classifier Epoch {epoch+1}: loss = {avg_loss:.4f}")
# Evaluate the classifier on test data.
# Here we extract features from the RBM for each test image.
if CLASSIFIER != 'LogisticRegression':
classifier.eval()
correct = 0
total = 0
features_list = []
labels_list = []
with torch.no_grad():
for images, labels in test_loader:
v = images.view(-1, 784).to(device)
# Extract hidden activations; you can use either h_prob or h_sample.
h_prob, _ = rbm.sample_h(v)
if CLASSIFIER == 'LogisticRegression':
features_list.append(h_prob.cpu().detach().numpy())
labels_list.append(labels.numpy())
else:
outputs = classifier(h_prob)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted.cpu() == labels).sum().item()
if CLASSIFIER == 'LogisticRegression':
test_features = np.concatenate(features_list)
test_labels = np.concatenate(labels_list)
predictions = classifier.predict(test_features)
accuracy = accuracy_score(test_labels, predictions) * 100
macro_f1 = f1_score(test_labels, predictions, average="macro")
else:
accuracy = 100 * correct / total
all_preds = []
all_labels = []
classifier.eval()
with torch.no_grad():
for images, labels in test_loader:
v = images.view(-1, 784).to(device)
h_prob, _ = rbm.sample_h(v)
outputs = classifier(h_prob)
_, predicted = torch.max(outputs.data, 1)
all_preds.extend(predicted.cpu().numpy())
all_labels.extend(labels.numpy())
macro_f1 = f1_score(all_labels, all_preds, average="macro")
print(f"Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}")
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1)
mlflow.end_run()
return float(accuracy if accuracy is not None else 0.0)
if __name__ == "__main__":
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=1) # n_trials set to 1 for quick rendering
print(study.best_params)
print(study.best_value)
print(study.best_trial)
(Figure 13)
What the plot shows:
When using RBM-extracted hidden features as input to logistic regression, the inverse regularization strength does not appear to be a strong predictor of test accuracy.
(Figure 14)
What the plot shows:
Optuna slightly favors higher number of hidden units in the rbm with a peak at 5340 (and similar peaks 5358, 5341, etc.). However, after 7000 units, accuracy appears to decline suggesting the optimum number of units was reached around that 5300 mark.
Model 5
Model 5: Feed Forward Network on RBM Hidden Features (of Fashion MNIST Data)
Click to Show Code and Output
from sklearn.metrics import accuracy_score, f1_score
CLASSIFIER = 'FNN'
if CLASSIFIER == 'LogisticRegression':
experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-withrbm")
else:
experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-withrbm")
class RBM(nn.Module):
def __init__(self, n_visible=784, n_hidden=256, k=1):
super(RBM, self).__init__()
self.n_visible = n_visible
self.n_hidden = n_hidden
# Initialize weights and biases
self.W = nn.Parameter(torch.randn(n_hidden, n_visible) * 0.1)
self.v_bias = nn.Parameter(torch.zeros(n_visible))
self.h_bias = nn.Parameter(torch.zeros(n_hidden))
self.k = k # CD-k steps
def sample_h(self, v):
# Given visible v, sample hidden h
p_h = torch.sigmoid(F.linear(v, self.W, self.h_bias)) # p(h=1|v)
h_sample = torch.bernoulli(p_h) # sample Bernoulli
return p_h, h_sample
def sample_v(self, h):
# Given hidden h, sample visible v
p_v = torch.sigmoid(F.linear(h, self.W.t(), self.v_bias)) # p(v=1|h)
v_sample = torch.bernoulli(p_v)
return p_v, v_sample
def forward(self, v):
# Perform k steps of contrastive divergence starting from v
v_k = v.clone()
for _ in range(self.k):
_, h_k = self.sample_h(v_k) # sample hidden from current visible
_, v_k = self.sample_v(h_k) # sample visible from hidden
return v_k # k-step reconstructed visible
def free_energy(self, v):
# Compute the visible bias term for each sample in the batch
vbias_term = (v * self.v_bias).sum(dim=1) # shape: [batch_size]
# Compute the activation of the hidden units
wx_b = F.linear(v, self.W, self.h_bias) # shape: [batch_size, n_hidden]
# Compute the hidden term
hidden_term = torch.sum(torch.log1p(torch.exp(wx_b)), dim=1) # shape: [batch_size]
# Return the mean free energy over the batch
return - (vbias_term + hidden_term).mean()
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.FashionMNIST(root='./data', train=True, transform=transform, download=True)
test_dataset = datasets.FashionMNIST(root='./data', train=False, transform=transform, download=True)
def objective(trial):
num_rbm_epochs = trial.suggest_int("num_rbm_epochs", 5, 5)# 24, 33)
batch_size = trial.suggest_int("batch_size", 192, 1024)
rbm_lr = trial.suggest_float("rbm_lr", 0.05, 0.1)
rbm_hidden = trial.suggest_int("rbm_hidden", 384, 8192)
mlflow.start_run(experiment_id=experiment.experiment_id)
if CLASSIFIER != 'LogisticRegression':
fnn_hidden = trial.suggest_int("fnn_hidden", 192, 384)
fnn_lr = trial.suggest_float("fnn_lr", 0.0001, 0.0025)
mlflow.log_param("fnn_hidden", fnn_hidden)
mlflow.log_param("fnn_lr", fnn_lr)
num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)# 40, 60)
mlflow.log_param("num_rbm_epochs", num_rbm_epochs)
mlflow.log_param("batch_size", batch_size)
mlflow.log_param("rbm_lr", rbm_lr)
mlflow.log_param("rbm_hidden", rbm_hidden)
mlflow.log_param("num_classifier_epochs", num_classifier_epochs)
# Instantiate RBM and optimizer
device = torch.device("mps")
rbm = RBM(n_visible=784, n_hidden=rbm_hidden, k=1).to(device)
optimizer = torch.optim.SGD(rbm.parameters(), lr=rbm_lr)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
rbm_training_failed = False
# Training loop (assuming train_loader yields batches of images and labels)
for epoch in range(num_rbm_epochs):
total_loss = 0.0
for images, _ in train_loader:
# Flatten images and binarize
v0 = images.view(-1, 784).to(rbm.W.device) # shape [batch_size, 784]
v0 = torch.bernoulli(v0) # sample binary input
vk = rbm(v0) # k-step CD reconstruction
# Compute contrastive divergence loss (free energy difference)
loss = rbm.free_energy(v0) - rbm.free_energy(vk)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch+1}: avg free-energy loss = {total_loss/len(train_loader):.4f}")
if np.isnan(total_loss):
rbm_training_failed = True
break
if rbm_training_failed:
accuracy = 0.0
macro_f1 = 0.0
print("RBM training failed — returning 0.0 for accuracy and macro F1")
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1)
mlflow.set_tag("status", "rbm_failed") # Optional tag
mlflow.end_run()
return float(accuracy)
else:
rbm.eval() # set in evaluation mode if using any layers that behave differently in training
features_list = []
labels_list = []
for images, labels in train_loader:
v = images.view(-1, 784).to(rbm.W.device)
v = v # (optionally binarize or use raw normalized pixels)
h_prob, h_sample = rbm.sample_h(v) # get hidden activations
features_list.append(h_prob.cpu().detach().numpy())
labels_list.append(labels.numpy())
train_features = np.concatenate(features_list) # shape: [N_train, n_hidden]
train_labels = np.concatenate(labels_list)
# Convert pre-extracted training features and labels to tensors and create a DataLoader
train_features_tensor = torch.tensor(train_features, dtype=torch.float32)
train_labels_tensor = torch.tensor(train_labels, dtype=torch.long)
train_feature_dataset = torch.utils.data.TensorDataset(train_features_tensor, train_labels_tensor)
train_feature_loader = torch.utils.data.DataLoader(train_feature_dataset, batch_size=batch_size, shuffle=True)
if CLASSIFIER == 'LogisticRegression':
# add optuna tuning same as log reg without RBM features...
lr_C = trial.suggest_float("lr_C", 0.01, 10.0, log=True)
mlflow.log_param("lr_C", lr_C) # Log the chosen C value
classifier = LogisticRegression(max_iter=num_classifier_epochs, C=lr_C, solver="saga")
classifier.fit(train_features, train_labels)
else:
classifier = nn.Sequential(
nn.Linear(rbm.n_hidden, fnn_hidden),
nn.ReLU(),
nn.Linear(fnn_hidden, 10)
)
# Move classifier to the same device as the RBM
classifier = classifier.to(device)
criterion = nn.CrossEntropyLoss()
classifier_optimizer = torch.optim.Adam(classifier.parameters(), lr=fnn_lr)
classifier.train()
for epoch in range(num_classifier_epochs):
running_loss = 0.0
for features, labels in train_feature_loader:
features = features.to(device)
labels = labels.to(device)
# Forward pass through classifier
outputs = classifier(features)
loss = criterion(outputs, labels)
# Backpropagation and optimization
classifier_optimizer.zero_grad()
loss.backward()
classifier_optimizer.step()
running_loss += loss.item()
avg_loss = running_loss / len(train_feature_loader)
print(f"Classifier Epoch {epoch+1}: loss = {avg_loss:.4f}")
# Evaluate the classifier on test data.
# Here we extract features from the RBM for each test image.
if CLASSIFIER != 'LogisticRegression':
classifier.eval()
correct = 0
total = 0
features_list = []
labels_list = []
with torch.no_grad():
for images, labels in test_loader:
v = images.view(-1, 784).to(device)
# Extract hidden activations; you can use either h_prob or h_sample.
h_prob, _ = rbm.sample_h(v)
if CLASSIFIER == 'LogisticRegression':
features_list.append(h_prob.cpu().detach().numpy())
labels_list.append(labels.numpy())
else:
outputs = classifier(h_prob)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted.cpu() == labels).sum().item()
if CLASSIFIER == 'LogisticRegression':
test_features = np.concatenate(features_list)
test_labels = np.concatenate(labels_list)
predictions = classifier.predict(test_features)
accuracy = accuracy_score(test_labels, predictions) * 100
macro_f1 = f1_score(test_labels, predictions, average="macro")
else:
accuracy = 100 * correct / total
all_preds = []
all_labels = []
classifier.eval()
with torch.no_grad():
for images, labels in test_loader:
v = images.view(-1, 784).to(device)
h_prob, _ = rbm.sample_h(v)
outputs = classifier(h_prob)
_, predicted = torch.max(outputs.data, 1)
all_preds.extend(predicted.cpu().numpy())
all_labels.extend(labels.numpy())
macro_f1 = f1_score(all_labels, all_preds, average="macro")
print(f"Test Accuracy: {accuracy:.2f}%")
print(f"Macro F1 Score: {macro_f1:.4f}")
mlflow.log_metric("test_accuracy", accuracy)
mlflow.log_metric("macro_f1", macro_f1)
mlflow.end_run()
return float(accuracy if accuracy is not None else 0.0)
if __name__ == "__main__":
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=1) # n_trials set to 1 for quick rendering
print(study.best_params)
print(study.best_value)
print(study.best_trial)
(Figure 15)
What the plot shows:
Highest accuracies cluster between 2000 and 4000 hidden units in the RBM with an outlier at 3764 hidden units. This possibly suggests too few hidden units lacks the complexity needed to explain the data; however, too many hidden units is perhaps causing some overfitting–resulting in poor generalization of the FNN classifier that receives the RBM hidden features.
(Figure 16)
What the plot shows:
Surprisingly, the number of hidden units in the FNN does not show a strong correlation with test accuracy. All hidden units tested seem to result in similar performance. This suggests the FNN is able to learn from the RBM features sufficently, and additional neurons do not significantly improve generalization.
| Model | Optuna Best Trial MLflow Test Accuracy(%) |
Macro F1 Score |
|---|---|---|
| Logistic Regression | 84.71 | 0.846 |
| Feed Forward Network | 88.06 | 0.879 |
| Convolutional Neural Network | 91.29 | 0.913 |
| Logistic Regression (on RBM Hidden Features) | 87.14 | 0.871 |
| Feed Forward Network (on RBM Hidden Features) | 86.95 | 0.869 |
(Table 2)